If we don't retain the v1alpha1 definitions, would it be possible to deploy the new version on top of an existing cluster without breaking anything? I can't seem to be able to apply the new CRDs without deleting the old ones first (The CustomResourceDefinition "kibanas.kibana.k8s.elastic.co" is invalid: spec.version: Invalid value: "v1beta1": must match the first version in spec.versions). Unless I have made a silly mistake, I assume that this means that people can't upgrade without some down time?

Also, should we consider implementing conversion to support a smoother upgrade flow?

I think conversion would require a webhook server which is currently removed in master due to #1723 But it is definitely something we would like to do once it is restored.

As to having multiple versions of the API:
I think we are still using the deprecated version attribute and we should be using versions instead which lists the different versions and indicates via served whether users can submit new objects using a particular version.

Can we try that and see if keeping v1alpha1 in the versions (but served: false) smoothens the upgrade process?

If we don't retain the v1alpha1 definitions, would it be possible to deploy the new version on top of an existing cluster without breaking anything?

No.

I assume that this means that people can't upgrade without some down time?

Yes.

In the current state it will be a breaking change that implies a downtime.

Can we try that and see if keeping v1alpha1 in the versions (but served: false) smoothens the upgrade process?

Yes, I'm going to try this to avoid the breaking.

As the schema has changed between v1alpha1 and v1beta1, I don't think we can have both versions referenced in the latest CRDs without implementing conversion.

I tried with:

  versions:
  - name: v1beta1
    served: true
    storage: true
  - name: v1alpha1
    served: false
    storage: false

And this workflow:

• deploy the operator 0.9
• deploy a test ES resource in v1alpha1
• apply the new CRDs with versions v1beta1 and v1alpha1
• update the operator to the latest version

Then the operator is failing with the following errors:

2019-09-25T14:50:26.314+0200 ERROR controller-runtime.eventhandler.EnqueueRequestForOwner Could not retrieve rest mapping {"ver": "0.10.0-SNAPSHOT-263b5431", "kind": "Elasticsearch.elasticsearch.k8s.elastic.co", "error": "no matches for kind "Elasticsearch" in version "elasticsearch.k8s.elastic.co/v1alpha1""}

2019-09-25T14:50:26.562+0200 ERROR controller-runtime.controller Reconciler error {"ver": "0.10.0-SNAPSHOT-263b5431", "controller": "elasticsearch-controller", "request": "default/test", "error": "Elasticsearch.elasticsearch.k8s.elastic.co "test" is invalid: []: Invalid value: map[string]interface {}{...}: validation failure list:\nspec.nodes.name in
body should match '[a-zA-Z0-9-]+'"}

Even if it's not our goal for now, I tried to update the test ES resource with the new schema (adding the name) and got:

2019-09-25T15:29:08.949+0200 ERROR controller-runtime.controller Reconciler error {"ver": "0.10.0-SNAPSHOT-263b5431", "controller": "elasticsearch-controller", "request": "default/test", "error": "Object default/test-es-http-certs-internal is already owned by another Elasticsearch controller test", "errorCauses": [{"error": "Object default/test-es-http-certs-internal is already owned by another Elasticsearch controller test"}]}

Would it help if instead of renaming we would keep the alpha versions around?

Ok I played around with this a bit here pebrc@76fddfc

and it looks like if we keep the original code around and fix the CRD generation (which was not running and broken) we can have both versions.

But:

• if served: true old resources will be reconciled by the new controller
• if served: false then controller-runtime.eventhandler.EnqueueRequestForOwner Could not retrieve rest mapping

Also:

• kubectl apply -f crds.yaml will fail miserably on update because the last-applied-configuration annoation created by kubectl will exceed the max annotation length
• our controller level version check should work once we update to the correct minVersion in the code

Well done!

• kubectl apply -f crds.yaml will fail miserably on update because the last-applied-configuration annoation created by kubectl will exceed the max annotation length

The description fields can be truncated or removed using crd:maxDescLen=[-1|0|n] (kubernetes-sigs/controller-tools#299).

-       $(CONTROLLER_GEN) crd paths="./pkg/apis/..." output:crd:artifacts:config=config/crds
+       $(CONTROLLER_GEN) crd:maxDescLen=100 paths="./pkg/apis/..." output:crd:artifacts:config=config/crds

The description fields can be truncated

I used kubectl replace instead which does not write the annotation

Just to be clear since the v1beta1 types are essentially just a copy and paste of the v1alpha1 types, and they need to be that way to be compatible between versions: if we make any type changes in the future we need to make sure we change both the v1alpha1 and v1beta1 types, correct? Until we have a conversion webhook that is.

if we make any type changes in the future we need to make sure we change both the v1alpha1 and v1beta1 types, correct?

No, if we make any type changes in the future, we need to create a new version (e.g. v1beta2 or v1).

Also, we use our controller-version to prevent the latest operator (1.0.0-beta1) to handle previous custom resources (0.9), to overcome the fact that we have no conversion webhook.

This should also close out #1770

Glad we are finally here. Thank you for your feedbacks.

Object default/test-es-http-certs-internal is already owned by another Elasticsearch controller test
I still don't know how to solve this error. The old resource's owner ref belongs to v1alpha1, but the new operator use v1beta1. I think version conversion cannot fix this. Is there any way to use new controllers to manage old version resources?